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1.  Introduction 


Training  and  education  tools  and  methods  must  be  of  sufficient  intelligence  to 
understand  the  needs  of  individual  learners  and  units  of  learners,  to  mitigate 
negative  learner  states,  and  to  guide  and  tailor  instruction  in  real-time  to  optimize 
learning.  These  tools  and  methods  must  also  be  affordable,  effective,  and  easy  to 
access  and  use.  These  requirements  are  enablers  of  the  US  Army  Learning  Model 
(ALM),  which  includes  an  emphasis  on  self-regulated  learning  (SRL)  where 
Soldiers  are  expected  to  manage  their  own  learning  and  career  development 
through  the  growth  of  metacognitive  (e.g.,  reflection),  self-assessment  and 
motivational  skills  (Butler  and  Winne  1995).  While  SRL  skills  are  difficult  to 
train  and  develop,  support  may  be  provided  to  the  learner  through  “adaptive 
training  technologies”  (tools  and  methods),  which  may  be  focused  to  guide 
learning  and  reinforce  SRL  principles. 

To  support  ALM,  the  US  Army  Research  Laboratory  (ARL)  has  developed  a 
program  of  research  called  “adaptive  training”,  which  includes  6  interdependent 
research  areas  or  vectors:  individual  learner  and  unit  modeling,  instructional 
management  principles,  domain  modeling,  authoring  tools  and  methods, 
evaluation  tools  and  methods,  and  architectural  and  ontological  support  for 
adaptive  training.  The  reports  documenting  these  vectors  expand  the  scope  of  the 
adaptive  tutoring  research  described  in  ARL-SR-0284  (Sottilare  2013)  to  support 
ALM  requirements  in  the  mid-term  and  long-term  evolution  of  training  and 
educational  technology:  the  Synthetic  Training  Environment  (STE)  and  the  Future 
Holistic  Training  Environment  for  Live  and  Synthetic  (FHTE-LS). 

This  report  (1  of  6  interdependent  research  outlines)  focuses  on  domain  modeling 
research  for  adaptive  training  and  education.  Today,  the  majority  of  intelligent 
tutoring  systems  (ITS’s) — a  form  of  adaptive  training  tool  to  support  one-to-one 
computer-based  instruction — support  well-defined  domains  in  mathematics, 
physics,  and  software  programming.  Since  Soldiers  operate  in  more  complex, 
dynamic,  and  ill-defined  domains,  it  is  necessary  to  expand  the  scope  of  adaptive 
training  tools  and  methods  to  support  training  and  education  in  these  militarily 
relevant  domains.  Domain  modeling  is  a  representation  of  knowledge  for  a 
particular  task  or  concept  and  includes:  domain  content  (a  library  of  scenarios, 
problem  sets,  or  knowledge  components);  an  expert  or  ideal  student  model  with 
measures  of  success  and  a  library  of  common  misconceptions;  and  a  library  of 
tactics  or  actions  (e.g.,  questions,  assessments,  prompts,  and  pumps)  that  can  be 
taken  by  the  tutor  to  engage  or  motivate  the  learner  and  optimize  learning. 
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2.  Research  Goals  and  Objectives 


The  goal  of  the  research  described  in  this  report  is  to  model  militarily  relevant 
training  domains  to  support  individually  tailored  and  intelligently  guided  training 
experiences  as  prescribed  by  the  US  Army  Learning  Model  (US  Army  Training 
and  Doctrine  Command  2011).  The  research  provides  guidelines,  best  practices, 
tools,  models,  and  methods  in  support  of  this  research  goal.  More  specifically,  we 
desire  to: 

•  Understand  and  model  the  characteristics,  similarities,  and  differences  of 
US  Anny  training  domains  (cognitive,  affective,  psychomotor,  social,  and 
hybrid)  with  respect  to  their  associated  knowledge  representations  to 
support  more  efficient  and  effective  authoring,  instruction,  and  evaluation 
of  adaptive  training  tools  and  methods. 

•  Understand  and  model  the  dimensions  (definition,  complexity,  and 
dynamics)  of  training  domain  representations  to  extend  the  capabilities  of 
traditional  ITS’s;  thereby,  support  challenging,  militarily  relevant  training 
domains. 

This  report  examines  the  background  and  requirements  for  adaptive  training 
capabilities  in  different  domains  along  with  research  challenges,  dimensions  of 
domain  modeling,  desired  end  states,  and  finally,  interdependencies  with  other 
adaptive  training  research  vectors. 

3.  Background 

While  human  tutoring  and  mentoring  are  common  teaching  tools,  current  US 
Anny  standards  for  training  and  education  are  group  instruction  and  classroom 
training —  also  known  as  one-to-many  instruction.  Group  instruction  and 
classroom  training  have  been  generally  focused  on  acquiring  and  applying 
knowledge  in  proxies  for  live  training  environments  (e.g.,  desktop  simulations, 
virtual  simulations,  constructive  simulations,  and  serious  games). 

Classroom  training,  especially  for  complex  topics,  is  often  taught  as  a  series  of 
lists  that  the  instructor  goes  through  in  a  linear  fashion  (Schneider  et  al.  2013). 
This  approach  puts  a  heavy  burden  on  the  learner  to  build  mental  models  and 
make  conceptual  connections.  Using  this  instructional  methodology  may  lead  to 
varying  degrees  of  success  due  to  individual  differences  in  skills,  traits,  and/or 
preferences.  More  complex,  ill-defined,  or  dynamic  tasks  may  be  difficult  to 
instruct  in  a  classroom  environment  especially  if  the  cognitive  elements  of  the 
task  require  spatial  interaction  to  develop/maintain  skills  (e.g.,  marksmanship). 
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Small  group  instruction  in  live  environments  has  also  been  used  to  assess 
application  of  knowledge  and  the  development  of  skills.  A  standard  feedback 
mechanism  for  US  Army  training  is  the  after-action  review  (AAR)  where 
significant  decision  points  and  actions  are  captured  for  small  group  discussion 
that  is  conducted  after  the  completion  of  a  training  event  to  help  capture  teachable 
moments  and  to  aid  Soldiers  in  reflecting  on  their  recent  training  experiences. 

Both  classroom  training  and  small  group  instruction  are  manpower  intensive; 
requiring  teachers,  mentors,  and  support  staff  to  guide  the  Soldier’s  experience. 
Today,  ITSs  primarily  guide  learner  training  and  education  for  cognitive  tasks  in 
well-defined  domains  (e.g.,  problem  solving  and  decision-making  tasks  in 
mathematics  and  physics).  Soldiers  tend  to  perform  cognitive,  affective, 
psychomotor,  and  social  tasks  in  both  well-defined  (e.g.,  building  clearing)  and 
ill-defined  domains  (e.g.,  leadership,  resource  allocation).  ITSs  generally  provide 
static  training  (e.g.,  sitting  at  a  desktop  computer  to  train  on  a  serious  game)  that 
falls  short  in  matching  the  dynamic  nature  of  many  US  Anny  operational  tasks 
(e.g.,  psychomotor  tasks);  and  thereby,  reducing  opportunities  to  develop  and 
transfer  skills  to  the  operational  environment. 

Research  is  needed  to  understand  the  characteristics,  similarities,  and  differences 
of  US  Army  training  domains  (i.e.,  cognitive,  affective,  psychomotor,  social,  and 
hybrid)  to  develop  efficient  and  effective  adaptive  training  and  educational  tools 
and  methods  that  support  self-regulated  learning  in  complex,  ill-defined,  and 
physically  dynamic  military  domains. 

3.1  Self-Regulated  Learning  and  the  US  Army  Learning  Model 

In  2011,  the  US  Army  placed  significant  emphasis  on  the  development  of  SRL 
skills  with  the  expectation  that  new  methods  of  instruction  (e.g.,  ITS’s)  would 
augment  institutional  training  (i.e.,  classroom  and  small  group  instruction).  One- 
to-one  human  tutoring  has  been  shown  to  be  significantly  more  effective  than 
one-to-many  instructional  methods  (e.g.,  traditional  classroom  instruction:  Bloom, 
1984;  VanLehn,  2011).  However,  it  is  not  practical  nor  is  it  affordable  to  have  1 
expert  human  tutor  to  mentor  each  Soldier  in  the  US  Anny  for  every  required 
operational  task.  This  alone  signals  the  need  for  capabilities  to  support  one-to- 
one,  tailored  training,  and  educational  experiences. 

Additionally,  under  the  ALM,  Soldiers  are  largely  responsible  for  managing  their 
own  learning,  but  SRL  skills  are  difficult  to  train  and  develop  (Butler  and  Winne 
1995;  Azevedo  et  al.  2009;  Graesser  and  McNamara  2010).  We  anticipate 
adaptive  training  tools  and  methods  will  fill  this  gap  and  will  provide  personalized 
guidance  to  acquire,  apply,  retain,  and  transfer  knowledge  and  skills  to  the 
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operational  environment.  This  signals  the  need  for  a  computer-regulated  learning 
strategy  to  augment  missing  SRL  skills;  however,  adaptive  training  technologies 
must  first  become  affordable,  sufficiently  adaptive,  and  easy  to  use  for  this 
strategy  to  be  realized. 

3.2  Motivation  for  Research 

A  promising  alternative  to  one-to-one  human  tutoring  is  one-to-one  adaptive 
training  tools  that  include  ITS’s.  Meta-analyses  and  reviews  support  the  claim  that 
ITS  technologies  routinely  improve  learning  over  classroom  teaching,  reading 
texts,  and/or  other  traditional  learning  methods.  These  meta-analyses  normally 
report  effect  sizes  (sigma  [a]),  which  refers  to  the  difference  between  the  ITS 
condition  and  a  control  condition  in  standard  deviation  units.  The  reported  meta¬ 
analyses  show  positive  effect  sizes  that  vary  from  a  =  0.05  (Dynarsky  et  al.  2007) 
to  a  =  1.08  (Dodds  and  Fletcher  2004),  but  most  hover  between  a  =  0.40  and  a  = 
0.80  (Ma  et  al.  in  press;  Fletcher  2003;  Graesser  et  al.  2012;  Steenbergen-Hu  and 
Cooper  2013,  2014;  VanLehn  2011).  Our  current  best  meta-meta  estimate  from 
all  of  these  meta-analyses  is  a  =  0.60.  This  performance  is  comparable  to  human 
tutoring,  which  varies  from  between  a  =  0.20  and  a  =  1.00  (Cohen  et  al.  1982; 
Graesser  et  al.  2011),  depending  on  the  expertise  of  the  tutor.  Human  tutors  have 
not  varied  greatly  from  ITS’s  in  direct  comparisons  between  ITS  and  trained 
human  tutors  (Olney  et  al.  2012;  VanLehn  2011;  VanLehn  et  al.  2007). 

Graesser  et  al.  (2015,  in  press)  are  convinced  that  some  subject  matters  will  show 
higher  effect  sizes  than  others  when  comparing  any  intervention  (e.g.,  computer 
trainers,  human  tutors,  group  learning)  to  a  control.  It  is  difficult  to  obtain  high- 
effect  sizes  for  literacy  and  numeracy  because  these  skills  are  ubiquitous  in 
everyday  life  and  habits  are  automatized.  For  example,  Ritter  et  al.  (2007) 
reported  that  the  Cognitive  Tutor  for  mathematics  has  shown  an  effect  size  of  a  = 
0.30-0.40  in  environments  with  minimal  control  over  instructors.  Human 
interventions  to  improve  basic  reading  skills  typically  report  an  effect  size  of  a  = 
0.20.  In  contrast,  when  the  student  starts  essentially  from  ground  zero,  such  as 
many  subject  matters  in  science  and  technology,  then  effect  sizes  are  expected  to 
be  more  robust.  ITS’s  show  effect  sizes  of  a  =  0.60-2.00  in  the  subject  matters  of 
physics  (VanLehn  2011;  VanLehn  et  al.  2005),  computer  literacy  (Graesser  et  al. 
2004;  Graesser  et  al.  2012),  biology  (Olney  et  al.  2012),  and  scientific  reasoning 
(Millis  et  al.  2011;  Halpern  et  al.  2012).  As  a  notable  example,  the  Digital  Tutor 
(Fletcher  and  Morrison  2012)  improves  infonnation  technology  by  an  effect  size 
as  high  as  a  =  3.70  for  knowledge  and  a  =  1.10  for  skills.  The  effect  size 
attributed  to  improved  instruction  and  improved  domain  knowledge  have  not  been 
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separated  in  this  analysis.  Such  large  effect  sizes  would  never  be  expected  in  basic 
literacy  and  numeracy. 

Overall,  these  are  promising  results  and  equate  to  an  increase  of  about  a  letter 
grade  improvement  over  traditional  classroom  instruction.  While  ITS’s  are  a 
promising  technology  to  support  adaptive  training  for  individuals  in  well-defined 
domains  like  mathematics,  physics,  and  computer  programming,  the  US  Anny 
requires  the  ability  to  develop  and  exercise  Soldier  skills  in  more  ill-defined 
domains  (e.g.,  leadership)  and  at  the  unit  level  (e.g.,  collaborative  learning  and 
team  training).  Developing  and  maintaining  the  ability  to  make  effective  decisions 
under  stress  and  in  complex  environments  is  also  desirable. 

Adaptive  systems  by  their  nature  require  additional  content  and  complexity  to 
support  tailored  learning  for  each  user  and  as  a  consequence  have  a  very  high 
development  cost,  a  major  barrier  to  adoption  by  the  US  Anny.  Adaptive  systems 
are  also  insufficiently  adaptive  to  support  tailored,  self-regulated  training  and 
educational  experiences  across  a  broad  spectrum  of  military  tasks  as  required  by 
the  ALM.  Today,  few  ITS  authoring  tools  are  generalized  across  all  of  the 
domains  requiring  training  and  no  evaluation  criteria  or  standards  have  been 
developed  to  promote  reuse  and  interoperability  among  ITS’s  (Sottilare  et  al. 
2012b).  In  other  words,  current  adaptive  systems  are  not  yet  intelligent  enough  to 
support  the  tailored  instruction  required  by  the  US  Anny  in  the  breadth  of 
domains  being  trained;  but  there  is  a  stable  foundation  of  50  years  of  science  on 
which  to  grow  an  adaptive  training  and  education  capability  for  the  US  Army. 

3.3  Adaptive  Training  and  Education  Definitions 

In  support  of  the  ALM  and  affordable  adaptive  training  and  educational 
capabilities  for  the  US  Army,  ARL  is  investigating  and  developing  adaptive  tools 
and  methods.  A  desired  end-state  is  the  automation  of  authoring  (creation) 
processes,  instruction,  and  evaluation  of  computer-regulated  training  and 
education  capabilities  to  help  build  SRL  skills  and  support  mixed-initiative 
interaction.  A  major  goal  within  this  research  program  is  to  reduce  the  time/cost 
and  knowledge/skill  required  to  author,  deliver,  and  evaluate  adaptive 
technologies  to  make  them  usable  by  a  larger  segment  of  the  US  Anny  training 
and  educational  community. 

Adaptive  training  and  education  research  includes  elements  of  adaptive  tutoring, 
distributed  learning,  virtual  humans,  and  training  effectiveness  evaluation.  For 
additional  detail  on  research  specific  to  ITS’s,  refer  to  ARL-SR-0284  (Sottilare 
2013).  Definitions  are  provided  for  this  section  to  distinguish  between  adaptive 
training  and  education  elements  and  also  to  highlight  their  relationships: 
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Adaptive  Tutoring  -  also  known  as  intelligent  tutoring;  tailored  instructional 
methods  to  provide  one-to-one  and  one-to-many  computer-guided  experiences 
focused  on  optimizing  learning,  comprehension,  perfonnance,  retention, 
reasoning,  and  transfer  of  knowledge  and  acquired  skills  to  the  operational 
environment. 

Adaptive  Tutoring  Systems  -  also  known  as  ITS’s;  the  mechanism  or 
technologies  (tools  and  methods)  to  provide  tailored  training  and  educational 
experiences;  adaptive  tutoring  systems  respond  to  changing  states  in  the  learner 
and  changing  conditions  in  the  training  environment  to  optimize  learning; 
adaptive  tutoring  systems  anticipate  and  recognize  teachable  moments. 

Virtual  Humans  -  arti  ficial  1  y-i ntcl  1  i gent  visual  representations  of  people  that 
simulate  or  emulate  cognitive,  affective,  physical,  and  social  processes. 

Distributed  Learning  -  concurrent  distribution  of  training  and  educational 
content  to  multiple  users  at  the  point-of-need  in  which  content  is  intelligently 
selected  to  support  learning,  increased  performance,  and  long-term  competency  in 
selected  domains. 

Training/Learning  Effectiveness  -  evaluation  of  the  impact  of  training  and 
educational  tools  and  methods  on  usability,  learning,  comprehension, 
perfonnance,  retention,  reasoning,  and  transfer  of  knowledge  and  acquired  skills 
to  the  operational  environment. 

Adaptive  Training  and  Education  Systems  -  a  convergence  of  ITSs  and 
external  training  and  education  capabilities  (e.g.,  serious  games,  virtual  humans, 
simulations)  to  support  engaging  experiences  with  reduced  need  for  authoring 
(Sottilare  2015). 

Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  (Sottilare  et  al.  2012a; 
Sottilare  et  al.  2013a)  -  an  open-source,  modular  architecture  whose  goals  are  to 
reduce  the  cost  and  skill  required  for  authoring  adaptive  training  and  educational 
systems,  to  automate  instructional  delivery  and  management,  and  to  develop  and 
standardize  tools  for  the  evaluation  of  adaptive  training  and  educational 
technologies. 

Adaptive  training  and  education  research  at  ARL  is  being  conducted  across  6 
interdependent  research  vectors:  individual  learner  and  unit  modeling; 
instructional  management  principles;  domain  modeling,  authoring  tools  and 
methods;  evaluation  tools  and  methods;  and  architectural  and  ontological  support. 
This  report  (1  of  6  interdependent  research  outlines)  focuses  on  domain  modeling 
research  for  adaptive  training  systems  with  the  goal  of  guiding  learning  in 
militarily  relevant  training  and  educational  domains. 
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Soldiers  operate  in  a  variety  of  complex,  dynamic,  ill-defined  domains  where 
their  ability  to  persevere  in  the  face  of  adversity,  adapt  to  their  situation, 
collaborate,  and  think  critically  are  key  to  the  successful  completion  of  their 
assigned  missions.  In  order  to  develop  and  exercise  these  skills,  it  is  paramount 
for  Soldiers  to  train  in  challenging  environments.  Presently  these  few  challenging 
training  environments  have  been  largely  provided  through  manpower-intensive 
methods  or  systems  with  little  ability  to  adapt  instruction  to  support  their  learning 
needs.  To  illustrate  this  point  Franke  (2011)  asserts  that  through  the  use  of  case 
study  examples,  instruction  can  provide  the  pedagogical  foundation  for  decision¬ 
making  under  uncertainty.  However,  this  approach  is  limited  in  implementation 
by  the  expanse  of  potential  cases  that  would  need  to  be  consistently  updated  and 
maintained  to  support  large  populations  like  the  US  Army. 

As  noted  previously,  adaptive  systems  like  US’s  have  been  shown  to  be  effective 
in  promoting  learning  in  primarily  static  (e.g.,  learners  seated  at  desktop 
computers)  instructional  settings  within  relatively  simple,  well-defined  domains 
(e.g.,  mathematics,  physics)  for  individual  learners.  For  our  purposes,  static 
instruction  includes  cognitive,  affective,  or  social  training  tasks  where  a  desktop 
computer  delivers  instruction  and  where  the  physical  movement  of  the  learner  is 
limited  to  activities  that  can  be  conducted  while  seated.  For  example,  static 
instruction  can  effectively  support  cognitive  tasks  involving  decision-making  and 
problem-solving,  but  are  less  effective  for  training  tasks  involving  motion  and 
perception  (e.g.,  land  navigation  and  marksmanship).  Ideally,  we  desire  portable 
adaptive  instructional  capabilities  to  go  with  Soldiers  to  support  training  and 
education  at  their  point-of-need  across  a  wide  spectrum  of  US  Anny  operational 
tasks.  Research  is  needed  to  develop  tools  and  methods  to  support  broader  domain 
modeling,  which  is  representative  of  the  full  spectrum  of  US  Army  operational 
tasks.  Standards,  interoperability,  and  automation  (e.g.,  automated  scenario 
generation)  (Zook  et  al.  2012)  will  likely  play  a  significant  role  in  making 
adaptive  training  practical.  In  this  way  adaptive  training  technologies  will  have 
the  greatest  impact  on  organizational  learning  in  the  US  Army. 

4.  US  Army  Requirements  for  Adaptive  Training  Systems  and 
Domain  Modeling 

The  Army  Science  and  Technology  community  uses  Warfighter  Outcomes 
(WFOs)  as  the  authoritative  source  for  identifying  Warfighter  needs.  WFOs  are 
used  to  share  research  and  future  technology  solutions.  In  the  training  and 
education  (T&E)  domain,  the  adaptive  T&E  research  program  is  targeting  4 
specific  requirements  to  support  the  evolution  of  US  Army  training:  adaptive 


7 


training  and  education  systems;  big  data;  training  at  the  point-of-need;  and 
artificial  intelligence. 

4.1  Adaptive  Training  and  Education  Systems  and  Domain 
Modeling 

The  primary  gap  to  be  addressed  under  this  US  Anny  requirement  is  the  lack  of 
adaptive  systems  (e.g.,  intelligent  tutors)  to  support  individual  and  collective 
(team  or  unit)  training.  The  US  Army  needs  an  adaptive  training  and  education 
capability  that  is  persistent  and  easy  to  use/access  with  minimal  startup  time. 
There  are  also  requirements  to  automate  an  infonnal  AAR  (also  known  as  a 
postexercise  critique)  to  reduce  the  time  and  skills  needed  to  produce  the  AAR 
and  improve  its  focus  and  quality.  Another  line  of  thought  notes  that  the  artificial 
intelligence  in  ITS’s  could  be  used  to  facilitate  rapid  mission  planning  and  course- 
of-action  analyses  as  a  job  aid  in  operational  contexts. 

The  major  connection  between  the  adaptive  training  and  education  requirement 
and  the  domain  modeling  research  vector  is  the  need  to  extend  adaptive  system 
architectures  to  support  instruction  in  relevant  military  domains.  Affordable 
solutions  to  support  adaptive  training  in  more  complex,  ill-defined,  and  dynamic 
domains  will  more  closely  align  training  and  operations,  thereby  resulting  in  more 
efficient  transfer  of  knowledge  and  skills. 

4.2  Big  Data  and  Domain  Modeling 

The  primary  gap  to  be  addressed  under  this  US  Army  requirement  is  that  there  is 
a  lack  of  capability  to  handle  and  process  large  amounts  of  structured  and 
unstructured  data  (also  referred  to  as  big  data).  One  capability  needed  is  a 
structured  data  analytics  program  linking  individual  data  (e.g.,  achievements)  to 
required  long-tenn  competencies  in  military  occupational  specialties  (MOS’s). 
This  would  allow  Soldiers  to  understand  where  they  rank  in  terms  of  experiences 
and  achievements  among  other  Soldiers  in  their  MOS.  It  would  also  allow  the  US 
Army  to  identify  specific  experiences  among  successful  Soldiers  in  that  MOS  and 
provide  a  model  for  other  Soldiers  in  that  MOS  to  follow.  The  data  could  also  be 
used  by  course  managers  and  instructors  to  continuously  improve  instruction  and 
the  mental  models  of  both  human  and  computer-based  instructors.  Finally,  data 
collected  on  trainee  learning  and  performance  during  adaptive  training 
experiences  could  be  used  to  facilitate  Unit  Training  Management  (UTM)  where 
unit  commanders  would  have  access  to  empirical  data  to  support  unit  training 
decisions. 
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The  major  connection  between  the  US  Anny’s  big  data  requirement  and  domain 
modeling  is  ability  to  collect  learner  data,  learner  states,  and  training  environment 
data  to  make  the  connection  between  domain-specific  instructional  tactics  and 
their  varying  degrees  of  effectiveness  given  instructional  context  (existing 
conditions).  This  will  allow  course  managers  to  identify  best  practices  over  time 
and  to  promote  agile  configuration  management  of  instructional  content,  and 
effective  strategies,  tactics,  and  techniques. 

4.3  Training  at  the  Point-of-Need  and  Domain  Modeling 

The  primary  gap  to  be  addressed  under  this  US  Anny  requirement  is  the  lack  of 
an  easily  accessible,  persistent,  cost-effective,  and  low-overhead  training 
environment.  A  capability  is  needed  to  bring  training  to  Soldiers  instead  of 
Soldiers  going  to  fixed  training  locations.  This  point-of-need  training  capability 
would  be  easily  distributed,  web-based,  and  built  upon  open-enterprise 
architecture  in  the  cloud.  US  Anny  training  and  educational  opportunities  would 
be  available  on  demand  anywhere  and  anytime.  However,  it  should  be  noted  that 
the  delivery  mechanism  (e.g.,  laptop  computer,  mobile  device,  and  smart  glasses) 
for  adaptive  training  is  critical  in  determining  the  limitations  of  the  domain  model 
scope  and  complexity.  For  example,  it  may  be  extremely  difficult  to  train  all  the 
complexities  of  a  psychomotor  task  in  a  desktop  computer  setting. 

The  major  connection  between  point-of-need  training  and  domain  modeling  is  the 
practicality  of  extending  adaptive  training  beyond  the  desktop.  Low-cost 
commercial  tools  (e.g.,  smart  glasses)  must  be  investigated  to  detennine  their 
suitability  to  support  the  same  kinds  of  tutor-user  interaction  afforded  in  static 
desktop  applications.  The  cloud  architecture  to  support  adaptive  training  and 
education  will  be  required  to  operate  with  and  without  Internet  connectivity 
depending  on  the  location  of  the  learner  and  their  access  to  the  network.  For 
example,  if  a  Soldier  decides  to  take  a  2-hour  (h)  training  course  while  traveling 
and  knows  that  Internet  connectivity  will  be  intermittent,  he  might  decide  to 
download  the  course  to  his  device  and  take  it  offline.  The  architecture  must  be 
able  to  track  the  Soldier’s  progress  and  upload  results  when  connectivity  is  again 
available. 

4.4  Artificial  Intelligence  (Al)  Capabilities  and  Domain  Modeling 

The  primary  gap  to  be  addressed  under  this  US  Anny  requirement  is  that  the  US 
Anny  lacks  an  automated  capability  to  replicate  the  complexity  and  uncertainty  of 
the  operational  environment.  This  gap  specifically  points  to  the  lack  of 
adaptiveness  in  virtual  humans,  intelligent  tutoring  systems,  and  other  training 
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capabilities.  This  gap  leads  to  Soldiers  developing  training-response  strategies 
that  result  in  less  challenging  training  over  time  along  with  lower  engagement  and 
lower  levels  of  learning  and  transfer  of  skills  to  more  challenging  operational 
environments. 

The  major  connection  between  AI  capabilities  and  domain  modeling  involve  the 
discovery  and  innovation  of  techniques  to  support  a  concept  called,  “automated 
scenario  evolution”.  AI  capabilities  are  needed  to  support  automated  scenario 
evolution  where  AI  drives  the  generation  of  new  “child”  scenarios  from  a  single¬ 
parent  scenario  based  on  dimensions  of  that  scenario  and  the  state  of  the  trainee. 
In  this  way,  the  authoring  burden  for  highly  complex  training  and  educational 
domains  may  be  reduced. 

For  example,  consider  a  single  scenario  where  dimensions  include  variable 
challenge  levels  based  on  3  threats  (i.e.,  low,  moderate,  high),  3  types  of  field-of- 
view  (i.e.,  narrow,  moderate,  and  wide)  and  clear  line-of-sight  (i.e.,  near, 
moderate,  and  far).  AI  could  spawn  27  new  child  scenarios  based  on  combinations 
of  these  variables.  This  requirement  is  closely  linked  to  adaptive  training 
capabilities  described  in  Section  4.1  of  this  report  and  the  realization  of  this 
capability  will  enable  the  development  of  affordable  self-authoring  adaptive 
systems.  Through  this  capability,  complex  domains  may  be  modeled  for  adaptive 
training  systems  without  the  need  for  long  development  cycles  or  special 
authoring  skill  sets. 

Al-based  capabilities  in  adaptive  training  and  education  systems  may  also  support 
data  acquisition  (sensing),  natural  language,  problem-solving  strategies,  and 
perceptual/interaction  mechanisms  in  the  tutor. 

5.  Understanding  the  Dimensions  of  Domain  Modeling 

There  are  4  typical  elements  that  compose  ITSs,  a  prime  example  of  an  adaptive 
training  and  education  system:  a  learner  or  trainee  model,  an  instructional  or 
pedagogical  model,  a  domain  model,  and  some  type  of  user  interface.  The  domain 
model  typically  includes  an  expert  or  ideal  student  model  by  which  the  adaptive 
system  measures/compares/contrasts  the  progress  of  the  learner  toward  learning 
objectives.  The  domain  model  also  includes  the  training  environment,  the  training 
task,  and  all  of  the  associated  instructional  actions  (e.g.,  feedback,  questions, 
hints,  pumps,  and  prompts),  which  could  possibly  be  delivered  by  the  adaptive 
system  for  that  particular  training  domain.  Typical  interaction  between  the 
learner,  the  training  environment,  and  the  adaptive  system  (tutoring  agents)  is 
shown  in  Fig.  1. 
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Fig.  1  Adaptive  training  interaction 

Typical  training  systems  examine  the  interaction  between  the  learner  and  the 
training  environment  to  measure  progress  toward  learning  objectives.  The  learner 
acts  on  the  environment  (e.g.,  opens  a  door  or  makes  a  choice  to  move  into  the 
room  or  stay  outside)  and  then  observes  any  changes  or  reactions  within  the 
environment.  Adaptive  systems  add  a  layer  of  software-based  tutoring  agents  that 
are  designed  to  guide  the  learner  in  much  the  same  way  as  a  human  tutor  interacts 
with  a  learner.  The  tutoring  agents  observe  the  behaviors  of  the  learner  to  assess 
their  states  (e.g.,  performance  and  attitudes)  and  interact  with  the  learner  to 
provide  support,  direction,  and  instruction.  In  addition,  they  track  the  effect  of 
interactions  on  learning.  Tutoring  agents  also  interact  with  the  training 
environment  and  may  manipulate  the  environment  to  present  more  challenging  or 
less  challenging  scenarios  in  response  to  the  assessed  state  of  the  learner. 

One  method  to  describe  or  model  the  domain  is  by  the  type  of  task  that  is  being 
trained.  A  traditional  methodology  of  categorizing  tasks  is  Bloom’s  (Bloom  and 
Krathwohl  1956)  Taxonomy,  which  describes  hierarchically  ordered  skills 
(competency)  in  the  cognitive  task  domain.  Due  to  the  contributions  of  others, 
Bloom’s  original  taxonomy  has  evolved  over  time  to  include  an  affective 
(Anderson  and  Krathwohl  2001),  psychomotor  (Simpson  1972),  and  a  social 
hierarchy  (Soller  2001;  Sottilare  et  al.  2011).  The  following  sections  describe 
each  task  domain  and  its  relationship  to  adaptive  training  and  education. 
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5.1  Cognitive  Domain 


Sometimes  called  the  “thinking”  domain,  tasks  in  this  domain  stress  the  learner’s 
thinking  capacity  (i.e.,  working  memory,  executive  control,  workload 
management,  multitasking),  problem-solving  and  planning,  decision-making, 
comprehension,  reasoning,  and  attentional  focus  or  engagement.  The 
detennination  of  cognitive  skills  may  be  based  on  learner  behaviors  to  indicate 
increases  in  complex  and  abstract  mental  capabilities  (Anderson  and  Krathwohl 
2001). 

A  revision  of  Bloom’s  Taxonomy  (Anderson  and  Krathwohl  2001)  tracks  a  series 
of  behaviors  from  low  to  high  cognitive  skills:  1)  remembering  -  the  learner’s 
ability  to  recognize  and  recall  information,  2)  understanding  (also  known  as 
comprehension)  -  the  learner’s  ability  to  organize,  compare,  and  interpret 
information,  3)  applying  -  the  learner’s  ability  to  use  infonnation  to  solve 
problems,  4)  analyzing  -  the  learner’s  ability  to  examine  information  and  make 
inferences  from  that  information,  5)  evaluating  -  the  learner’s  ability  to  use 
information  to  make  optimal  judgments,  and  6)  creating  -  learner’s  ability  to 
build  new  models  (e.g.,  plans)  from  information. 

Most  of  the  ITS’s  in  existence  today  focus  on  the  cognitive-task  domain 
(Anderson  et  al.  1995;  Ritter  et  al.  2007;  Graesser  et  al.  2012).  Examples  include 
model-tracing  (also  called  example  tracing)  tutors  that  use  a  set  of  steps  to  walk 
the  learner  through  the  process  of  solving  a  problem  (Koedinger  et  al.  2012). 
Mathematics,  physics,  and  software  programming  are  the  most  common  types  of 
model-tracing  tutors.  These  domains  constitute  simple  procedural  tasks  and  are 
usually  rule-based. 

Matthews  (2014)  notes  organizations  generally  do  a  good  job  of  training 
relatively  simple  skills.  However,  a  more  challenging  goal  is  to  teach  higher  order 
cognitive  skills  such  as  decision-making  and  judgment.  The  US  Army  has  large 
investments  in  partial-task  and  scenario-based  training  systems  that  use  relatively 
fixed  strategies  to  guide  the  learner  based  primarily  on  individual  and  team 
perfonnance  measures.  A  concern  with  these  systems  is  that  Soldiers  learn  how  to 
win  within  the  constraints  of  the  system  but  the  effect  on  retention  and  transfer  is 
not  well  understood.  Research  is  needed  to  build  adaptivity  into  these  training 
systems  and  thereby  optimize  deep  learning.  A  goal  of  this  research  is  to  reduce 
the  time  to  competency  to  allow  time  for  automaticity  through  overtraining  and 
deeper  learning  experiences  that  transfer  to  the  operational  environment. 
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5.2  Affective  Domain 


Sometimes  called  the  “feeling”  domain,  tasks  in  this  domain  are  intended  to 
develop  emotional  intelligence  or  skills  in  self-awareness  and  growth  in  attitudes, 
emotion,  and  feelings.  The  goal  is  to  manage  emotions  in  positive  ways  to  relieve 
stress,  communicate  effectively,  empathize  with  others,  overcome  challenges,  and 
defuse  conflict  (Goleman  2006).  While  listed  as  separate  domain,  affect  has  an 
interdependent  relationship  with  cognition  and  learning.  Specifically,  confusion, 
frustration,  boredom,  surprise,  delight,  flow,  and  anxiety  are  considered  major 
moderators  of  learning  (D’Mello  2013;  Pekrun  2006).  For  example,  cognitive 
readiness — the  capability  to  maintain  perfonnance  and  mental  well-being  in 
complex,  dynamic,  unpredictable  environments  that  may  elicit  affective 
responses.  Dimensions  of  cognitive  readiness,  according  to  Kluge  and  Burkolter 
(2013),  include  concepts  such  as  risk-taking  behavior,  emotional  stability  and 
coping,  which  may  be  considered  part  of  the  affective  domain. 

A  revision  of  Bloom’s  Taxonomy  (Anderson  and  Krathwohl  2001)  tracks  a  series 
of  behaviors  from  low-affective  state  to  high:  1)  receiving  -  the  learner  takes  in 
infonnation,  2)  responding  -  the  learner  takes  in  infonnation  and  responds/reacts, 
3)  valuing  -  the  learner  attaches  value  to  infonnation,  4)  organizing  -  the  learner 
sorts  information  and  builds  mental  models,  and  5)  characterizing  -  the  learner 
matches  mental  models  to  values  and  beliefs  ultimately  influencing  (e.g., 
promoting  or  limiting)  the  learner’s  behavior. 

Very  little  training  (outside  of  classroom-based  training)  is  currently  provided  to 
exercise/grow  skills  in  this  important  task  domain  and  almost  no  adaptive  training 
has  been  created  to  support  this  domain.  However,  D’Mello  and  Graesser  (2012) 
have  produced  an  affect-sensitive  tutor  that  exercises  emotional  intelligence  using 
the  AutoTutor  authoring  tools.  Research  is  needed  to  understand  measures  for  the 
affective  task  domain  and  any  unique  characteristics  required  to  author  affective 
domain  scenarios. 

5.3  Psychomotor  Domain 

Sometimes  called  the  “doing”  or  “action”  domain,  tasks  in  this  domain  are 
associated  with  physical  tasks  (e.g.,  marksmanship)  or  manipulation  of  a  tangible 
interface  (e.g.,  remotely  piloting  a  vehicle),  which  may  include  physical 
movement,  coordination,  and  the  use  of  the  motor-skills.  Development  of  motor- 
skills  requires  practice  and  is  measured  in  terms  of  speed,  precision,  distance, 
procedures,  or  techniques  during  execution  (Simpson  1972).  Simpson’s  hierarchy 
of  psychomotor  learning  ranges  from  low  to  high:  1)  perception  -  the  ability  to 
use  sensory  cues  to  guide  motor  activity;  2)  set  or  readiness  to  act;  3)  response  - 
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early  stages  of  learning  a  complex  skill  through  imitation  and  trial  and  error;  4) 
mechanism  -habitual  learned  responses;  5)  complex  overt  response  -  skillful 
perfonnance  of  complex  movements;  6)  adaptation  -  well-developed  skills  that 
are  modified  to  support  special  requirements;  and  7)  origination  -  the 
development  of  new  movement  patterns  to  fit  unique  situations. 

While  this  domain  is  well  represented  in  US  Anny  training,  research  is  needed  to 
build  adaptiveness  into  these  training  systems  and  thereby  optimize  deep  learning. 
Again,  a  goal  of  this  research  is  to  reduce  the  time  to  competency  to  allow  time 
for  automaticity  through  overtraining  and  deeper  learning  experiences  that 
transfer  to  the  operational  environment. 

5.4  Social  Domain 

Sometimes  called  the  “collaborative”  domain,  tasks  in  this  domain  include  a  set  of 
collaborative  characteristics  or  measures  of  learning  in  the  social  domain  as 
defined  by  Soller  (2001):  1)  participation,  2)  social  grounding  -  team  members 
“take  turns  questioning,  clarifying  and  rewording  their  peers’  comments  to  ensure 
their  own  understanding  of  the  team’s  interpretation  of  the  problem  and  the 
proposed  solutions”,  3)  active  learning  conversation  skills  -  quality 
communication,  4)  performance  analysis  and  group  processing  -  groups 
discuss  their  progress,  and  decide  what  behaviors  to  continue  or  change  (Johnson 
et  al.  1990)  and  5)  promotive  interaction  -  also  known  as  “win-win”  this 
characteristic  occurs  when  members  of  a  group  perceive  that  they  can  only  attain 
their  goals  if  their  team  members  also  attain  their  goals. 

This  domain  is  different  from  the  cognitive,  affective,  and  psychomotor  domain  in 
that  Soller’ s  collaborative  learning  skills  taxonomy  is  multilayered:  skills, 
subskills,  and  attributes/behaviors  as  shown  below: 

•  Conversation  Skill 

■  Task  subskill:  Coordinate  group  process,  request  focus  change, 
summarize  information,  and  end  participation  behavior 

■  Maintenance  subskill:  Request  attention,  suggest  action,  request 
confirmation,  listening  behavior,  and  apologize  behaviors 

■  Acknowledge  subskill:  Appreciation,  accept/confirm,  and  reject 
behaviors 

•  Active  Learning  Skill 

■  Request  subskill:  Information-seeking,  elaboration,  clarification, 
justification,  opinion-seeking,  and  illustration  behaviors 
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■  Inform  subskill:  Rephrase,  lead,  suggest,  elaborate,  explain/clarify, 
justify,  and  assert  behaviors 

■  Motivate  subskill:  Encourage  and  reinforce  behaviors 
•  Creative  Conflict  Skill 

■  Argue  subskill:  Conciliate,  agree,  disagree,  offer  alternatives,  infer, 
suppose,  propose  exception,  and  doubt  behaviors 

■  Mediate  sub  skill:  Teacher  mediation 

The  Program  for  International  Student  Assessment  (PISA)  (2015)  is  a  worldwide 
study  by  the  Organization  for  Economic  Cooperation  and  Development  (OECD) 
in  both  member  and  nonmember  nations.  This  study  focuses  on  15-year-old 
students  and  their  scholastic  performance  in  mathematics,  science,  and  reading. 
During  PISA  2015,  OECD  defined  a  matrix  of  collaborative  problem-solving 
skills  (Table  1).  Some  of  the  skills  and  associated  behaviors  in  this  matrix  may 
also  apply  to  situational  problem  solving  in  the  US  Army  (e.g.,  staff-level 
organizations  evaluating  options  to  meet  objectives  during  military  operations). 
Research  is  needed  to  determine  if  the  model  in  this  matrix  will  generalize  beyond 
its  original  application  to  15-year-old  students. 

Table  1  Matrix  of  collaborative  problem  solving  skills 


(1)  Establishing  and 
maintaining  shared 
understanding 

(2)  Taking  appropriate 
action  to  solve  the  problem 

(3)  Establishing  and 
maintaining  team 
organisation 

(A)  Exploring  and 
Understanding 

(A1)  Discovering 
perspectives  and  abilities  of 
team  members 

(A2)  Discovering  the  type  of 
collaborative  interaction  to 
solve  the  problem,  along  with 
goals 

(A3)  Understanding  roles  to 
solve  problem 

(B)  Representing  and 
Formulating 

(B1)  Building  a  shared 
representation  and 
negotiating  the  meaning  of 
the  problem  (common 
ground) 

(B2)  Identifying  and 
describing  tasks  to  be 
completed 

(B3)  Descnbe  roles  and  team 
organisation  (communication 
protocol/rules  of  engagement) 

(C)  Planning  and 
Executing 

(Cl)  Communicating  with 
team  members  about  the 
actions  to  be/  being 
performed 

(C2)  Enacting  plans 

(C3)  Following  rules  of 
engagement,  (e  g.,  prompting 
other  team  members  to 
perform  their  tasks.) 

(D)  Monitoring  and 
Reflecting 

(D1)  Monitoring  and 
repainng  the  shared 
understanding 

(D2)  Monitoring  results  of 
actions  and  evaluating 
success  in  solving  the 
problem 

(D3)  Monitoring,  providing 
feedback  and  adapting  the 
team  organisation  and  roles 

Research  is  needed  to  create  and  apply  measures  for  both  collaborative  learning 
and  team  training  activities.  The  interdependent  nature  of  US  Army  tasks  also 
requires  tutoring  of  squads  and  other  echelons  of  teams  (collective  training). 
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Research  is  also  needed  to  develop  team  state  models  to  drive  adaptive  training 
decisions.  An  extensive  review  of  the  team  performance  and  tutoring  literature 
has  been  conducted  (Burke  et  al.  2015,  in  press)  to  determine  antecedents  of  team 
outcomes  (i.e.,  performance,  learning,  satisfaction,  and  viability),  which  include 
behavioral  measures  (Table  2),  along  with  models  of  cooperation  (Table  3),  and 
team  cognition  (Table  4). 

Table  2  Team  states  and  behavioral  measures 


Team 

Team 

Team 

Team 

Performance 

Learning 

Satisfaction 

Viability 

Team  Behaviors 

Communication 

YES 

YES 

YES 

YES 

Coordination 

YES 

YES 

YES 

YES 

Mutual  Support 

YES 

NOT 

YES 

TENTATIVE 

EXAMINED 

Reflexivity 

YES 

YES 

YES 

NOT 

EXAMINED 

Monitoring 

YES 

NOT 

NOT 

NOT 

EXAMINED 

EXAMINED 

EXAMINED 

Conflict 

YES 

TENTATIVE 

YES 

YES 

Task 

YES 

NOT 

YES 

YES 

EXAMINED 

Relationship 

YES 

NOT 

YES 

YES 

EXAMINED 

Transaction 

YES 

TENTATIVE 

TENTATIVE 

NOT 

EXAMINED 

Action 

YES 

TENTATIVE 

TENTATIVE 

NOT 

EXAMINED 

Interpersonal 

YES 

TENTATIVE 

NOT 

NOT 

EXAMINED 

EXAMINED 

Leadership 

YES 

NOT 

YES 

NOT 

EXAMINED 

EXAMINED 

Organizational 

YES 

NOT 

NOT 

NOT 

Citizenship 

Behaviors 

EXAMINED 

EXAMINED 

EXAMINED 

Conflict 

YES 

TENTATIVE 

YES 

TENTATIVE 

Management 
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Table  3  Team  states  and  measures  of  cooperation 


Team 

Team 

Team 

Team 

Performance 

Learning 

Satisfaction 

Viability 

Cooperation 

Trust 

YES 

YES 

YES 

NOT 

EXAMINED 

Collective 

YES 

NOT 

YES 

YES 

Efficacy 

EXAMINED 

Psychological 

YES 

TENTATIVE 

TENTATIVE 

TENTATIVE 

Safety 

Cohesion 

YES 

TENTATIVE 

YES 

TENTATIVE 

Justice 

TENTATIVE 

NOT 

NOT  EXAMINED 

NOT 

EXAMINED 

EXAMINED 

Table  4  Team  states  and  team  cognition  measures 


Team 

Team 

Team 

Team 

Performance 

Learning 

Satisfaction 

Viability 

Team 

Cognition 

Team  Mental 

YES 

NOT 

YES 

NOT 

Models 

EXAMINED 

EXAMINED 

Transactive 

YES 

NOT 

TENTATIVE 

YES 

Memory 

EXAMINED 

Systems 

Situation 

YES 

NOT 

NOT  EXAMINED 

NOT 

Awareness 

EXAMINED 

EXAMINED 

Blocks  in  each  table  indicate  either:  “YES”  to  represent  a  significant  effect  was 
found  between  1  of  the  antecedents  examined  in  the  literature  review  and  1  of  the 
team  outcomes;  “TENTATIVE”  to  represent  some  effect  or  conflicting  evidence 
was  found  between  1  of  the  antecedents  examined  and  1  of  the  team  outcomes;  or 
“NOT  EXAMINED”  to  represent  that  these  relationships  were  not  examined  as 
part  of  this  literature  review  and  meta-analysis.  The  “NOT  EXAMINED” 
category  represents  a  set  of  opportunities  for  future  research  in  team  tutoring. 

Additional  discussion  of  research  in  the  social  domain  may  be  found  on  team 
cognition  (Salas  and  Fiore  2004),  team  mental  models  (Fletcher  and  Sottilare 
2013;  Rouse  et  al.  1992),  and  situational  awareness  in  team  performance  (Salas  et 
al.  1995). 

6.  Domain  Modeling  Research  Goals  and  Challenges 

A  foundational  goal  of  adaptive  training  research  at  ARL  is  to  model  the 
perception,  judgment,  and  behaviors  of  expert  human  tutors  to  support  practical, 
effective,  and  affordable  learning  experiences  guided  by  computer-based  agents. 
To  this  end,  5  primary  goals  for  domain  modeling  for  adaptive  training  systems 


17 


have  been  identified  and  are  discussed  in  this  section  along  with  the  major 
challenges  or  barriers  to  success. 

6.1  Representing  and  Understanding  the  Influence  of  Domain 
Attributes 

Our  first  goal  is  to  conduct  research  to  detennine  the  influence  of  task  domain 
attributes  (e.g.,  complexity,  definition,  and  dynamics)  on  cognitive  mechanisms 
(e.g.,  learning,  comprehension,  perfonnance,  retention,  reasoning,  and  transfer  of 
knowledge  and  acquired  skills  to  the  operational  environment).  Research  is 
needed  to  analyze  related  and  unrelated  domains  to  determine  commonalities, 
performance  measures,  and  user  requirements. 

The  challenge  in  meeting  this  goal  is  that  since  each  domain  contains  unique 
domain  knowledge,  the  relationship  of  these  attributes  to  the  cognitive 
mechanisms  listed  previously  are  not  well  understood  across  various  task  domains 
(cognitive,  affective,  psychomotor,  social,  and  hybrid  tasks).  It  may  be  possible  to 
generalize  relationships  in  a  particular  task  domain  with  the  goal  being  to  identify 
attributes  with  high  effect  on  cognitive  mechanisms. 

Also  important  to  this  research  goal  is  the  ability  to  define  how  these  attributes  are 
measured,  how  qualitative  inputs  are  going  to  be  assessed  against  quantitative 
metrics,  and  how  stakeholder  requirements  and  learner-generated  content  (e.g., 
social  media  input  on  relevance  and  impact)  influences  the  modeling  of  each 
domain.  There  needs  to  be  a  matching  between  the  desired  requirements  and  what 
is  actually  authored  in  the  domain.  For  example,  tutors  for  marksmanship  for  the 
US  Anny  and  Marines  may  be  very  similar  based  on  the  task  domain 
(psychomotor)  but  may  also  have  different  instructional  methods  or  concepts 
based  on  organizational  requirements. 

6.2  Reducing  Time,  Cost,  and  Skill  to  Author  and  Deliver 
Instruction 

Our  second  goal  is  to  discover  and  improve  authoring  tools  and  methods  to  more 
easily  represent  domain  knowledge  including  methods  to  select  optimal 
instructional  tactics  (actions)  based  on  sound  instructional  strategies  (plans),  the 
instructional  context,  and  the  learner’s  states  and  traits  (Murray  1999;  Murray 
2003;  Sottilare  and  Gilbert  2011;  Murray  2015).  Specifically,  our  goal  is  to 
provide  sufficient  domain  knowledge  to  effectively  adapt  the  training  of  the  task 
for  individual  learners  and  units  while  significantly  lowering  the  cost  and  skills 
needed  to  author  adaptive  training  systems  for  the  US  Army. 
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Authoring  in  complex  domains  is  a  time  intensive  process  and  research  is  needed 
to  determine  what  attributes  of  the  domain  model  influence  successful  outcomes 
(e.g.,  skill  development)  and  antecedents  to  those  outcomes  (e.g.,  motivation, 
engagement,  and  grit),  which  may  influence  the  complexity,  definition,  and 
dynamics  of  the  domain  knowledge  (e.g.,  content,  feedback,  and  assessments) 
presented  to  the  learner  and  thereby  influence  the  cost. 

Adaptive  training  systems  by  their  nature  offer  more  flexibility  and  are  tailored  to 
individual  learners.  Given  the  variability  of  learner  attributes  across  the  general 
population,  this  creates  a  greater  demand  for  domain  authoring.  Finding  efficient 
methods  to  create  new  content  and  to  reuse  existing  content  (e.g.,  training  content 
in  existing  US  Army  training  simulations)  should  be  a  priority  for  domain 
modeling  research.  Specifically,  we  need  to  examine  methods  to  automate  large 
portions  of  the  authoring  process  including  the  automated  development  of  expert 
models  (sometimes  called  ideal  student  models),  the  automated  generation  of 
scenario  variants  from  base  cases,  and  the  authoring  of  assessments  from  which 
the  ITS  detennines  progress  and  corresponding  strategies  and  tactics. 

Additionally,  our  goal  is  to  discover/improve  methods  to  provide  adaptive 
instruction  that  is  easily  accessible  and  is  available  at  the  Soldier’s  point-of-need 
(anyplace)  and  anytime  (24/7/365). 

6.3  Improving  the  Interoperability  of  Domain  Models 

Our  third  goal  is  to  develop  standards  for  interoperability  to  promote  reuse  of 
domain  knowledge  (e.g.,  content,  expert  models,  question  banks,  assessments,  and 
tactics).  Specifically,  our  goal  is  to  set  interoperability  standards  for  US  Anny 
training  resulting  in  reduced  development  and  maintenance  costs  and  increased 
speed  of  adoption  for  adaptive  training  technologies.  The  major  challenges  are 
that  no  standards  exist  for  adaptive  training  and  interoperability — the  ability  to 
pull  and  replace  one  model  or  domain  element  with  another  authored  elsewhere — 
is  extremely  low.  Effort  should  be  expended  to  work  with  the  ITS  community  to 
develop  interoperability  standards  to  maximize  reuse  of  domain  knowledge. 
Defining  standard  methods  of  representing  and  interfacing  with  domain 
knowledge  will  allow  a  single  adaptive  training  architecture  or  framework  to 
support  multiple  domains  and  reduce  authoring  costs.  This  is  the  primary  driver 
for  the  development  of  GIFT. 

The  US  Anny’s  focus  on  scenario-based  training  along  with  the  significant 
investment  in  training  infrastructure  should  also  be  considered  in  developing 
interoperability.  To  this  end,  GIFT  includes  a  standard  interface  specification,  or 
gateway,  so  that  US  Army  training  systems  that  meet  this  specification  can  be 
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integrated  to  provide  adaptive  training  capabilities.  In  addition,  the  GIFT  gateway 
currently  has  an  Institute  of  Electrical  and  Electronics  Engineers  (IEEE)  1278 
Distributed  Interactive  Simulation  (DIS)  compliant  interface  to  receive  tactical 
data  (e.g.,  entity  location)  to  support  instructional  decisions  and  to  push 
instructional  tactics  (e.g.,  interaction  with  the  learner  or  changes  to  the  training 
environment)  to  DIS-compatible  training  simulations. 

6.4  Optimizing  the  Selection  of  Tactics 


Our  fourth  goal  is  to  optimize  the  selection  of  tactics — domain-specific  actions  by 
the  tutor — to  provide  the  greatest  opportunity  for  performance,  learning,  retention, 
and  transfer.  In  GIFT,  tactics  are  the  actions  taken  by  the  tutor  in  response  to 
learner  states  and  instructional  context  (e.g.,  conditions  of  the  scenario  or  problem 
presented),  as  shown  in  Fig.  2  and  are  constrained  by  available  options  provided 
during  the  authoring  process.  Improving  the  usability  and  efficiency  of  authoring 
tools  will  likely  result  in  a  greater  number  of  available  options  for  adaptive 
training  domains. 


individual  learner  measures 

•  behavioral  &  physiological 

•  traits  &  preferences 

•  achievements 


individual  learner  states 

•  cognition  (engagement) 

•  affect  (motivation) 

•  physical  (fatigue) 

•  knowledge  and  skill 

•  performance 

•  retention  and  transfer 


measures  &  states 


i 


tutor  planning  and  action  processes 

i 


learner 

informs 

learner 

influence  instructional 

influences 

instructional 

data 

* 

states 

selection 

selection 

instructional 

context 


t 


modify  scenario  or  problem 


interact  with  learner  (e  g.,  assessments,  feedback,  encouragement,  reflective  prompts) 


Fig.  2  Updated  individual  learning  effect  model 

Unlike  instructional  strategies,  which  are  derived  from  good  pedagogical  practices 
based  on  learning  theory  and  influenced  by  the  learner’s  states,  tactics  are 
domain-specific  actions  by  the  tutor  and  may  not  be  generalized  across  all  task 
domains.  Research  is  needed  to  determine  methods  to  select  the  best  possible 
tactic  given  the  selected  instructional  strategy,  the  training  domain,  and  the 
availability  of  tactics. 
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Modeling  the  expert  behaviors  of  human  tutors  may  be  a  starting  point,  but 
accurate  assessment  methods  are  needed  for  both  individual  and  team-level  states. 
These  states  are  critical  in  selecting  appropriate  strategies  (plans  for  action)  and 
tactics  (actions:  e.g.,  assessments,  feedback,  questions,  changes  to  the  training 
environment)  per  the  Learning  Effect  Model  (Sottilare  2012;  Fletcher  and 
Sottilare  2013;  Sottilare  2013;  Sottilare  et  al.  2013b)  as  updated  for  both 
individuals  (Fig.  2)  and  teams  (Fig.  3).  Assessment  of  team  states  may  also  be 
useful  in  determining  constraints  to  be  monitored  by  tutoring  agents  and 
interactions  with  the  learner  and  the  training  environment  as  shown  earlier  in 
Fig.  1. 


individual  learner  states 

•  cognition  (engagement)  team  or  shared  states 

individual  learner  measures  •  affect  (motivation)  •  shared  cognition 

•  behavioral  &  physiological  •  physical  (fatigue)  •  shared  affect 

•  traits  &  preferences  •  knowledge  and  skill  •  team  knowledge  and  skill 

•  achievements  •  performance  •  team  performance 

•  retention  and  transfer  potential 

learner#1  informs  _  learner #1 

data  *  states 


measures  &  states 


tutor  planning  and  action  processes 


learner#2 

data 


learner#3 

data 


learner#n 

data 


Fig.  3  Updated  team  learning  effect  model 

6.5  Extending  Adaptive  Training  to  Militarily  Relevant  Domains 

Our  fifth  goal  is  to  be  able  to  extend  adaptive  training  to  support  militarily 
relevant  domains.  Many  military  tasks  are  hybrids  of  task  domains  in  that  they 
include  aspects  of  cognitive  (thinking:  evaluating,  problem-solving,  and  decision¬ 
making),  affective  (feeling:  making  value  judgments),  psychomotor  (doing: 
physical  action),  and/or  social  (collaborating:  working  in  teams).  US  Army 
training  differs  greatly  from  traditional  ITS’s,  which  are  primarily  problem-based 
(e.g.,  mathematics,  physics,  computer  programming)  and  generally  vary  only  in 
complexity.  Given  much  of  US  Army  training  is  scenario-based,  the  realism  of 
the  training  environment,  accessibility  of  the  training,  the  complexity  of  the 
scenario,  the  physical  dynamics  of  the  task,  and  the  variable  level  of  definition 
are  all  design  considerations  for  adaptive  training  systems  for  military  use.  It  will 
be  essential  to  match  the  attributes  of  the  environment  to  the  task  domain  by 
asking  the  question,  “What  is  necessary  to  train  the  task  effectively?”  This 
variability  in  adaptive  training  and  educational  domains  will  allow  for  greater 
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opportunities  for  Soldiers  to  train  at  the  point-of-need  and  to  train  more  closely  to 
how  they  fight.  This  is  anticipated  to  result  in  greater  learning,  performance, 
retention,  and  transfer  of  skills  to  the  operational  environment. 

As  in  all  training,  it  will  be  essential  to  match  the  realism  of  the  training 
environment  (e.g.,  serious  game,  virtual  simulation,  embedded  training)  to  enable 
learners  to  progress  toward  established  learning  objectives.  For  embedded  training 
where  Soldiers  bring  training  with  them  in  their  operational  platform  (e.g., 
ground,  air,  sea,  dismounted),  considerations  should  be  made  regarding  the  visual 
resolution  of  virtual  elements  of  the  training  environment  and  what  is  required  to 
train  the  task  (Sottilare  et  al.  2007).  For  example,  if  the  resolution  of  virtual 
targets  is  insufficient  to  support  either  detection  or  identification  of  the  targets  at 
comparable  distances  to  the  real  world  (also  known  as  live  environment),  then 
negative  training  may  result. 

Another  consideration  in  militarily  relevant  domains  is  accessibility.  Accessible 
learning  is  being  directly  addressed  by  the  research  and  development  of  GIFT 
(Sottilare  et  al.  2012a,  2012b;  Sottilare  et  al.  2013a),  an  adaptive  tutoring 
architecture  that  is  modular  and  service-oriented.  GIFT  will  also  support  access  to 
adaptive  tutoring  resources  (e.g.,  domain  content,  assessments,  Web  services)  via 
the  Internet  and  allow  content  to  be  presented  in  a  Web  browser. 

Adaptive  training  solutions  must  be  able  to  include  the  complexities  of  each  to 
provide  tailored  training  across  the  broad  spectrum  of  Soldier  tasks.  This  includes 
the  ability  to  align  more  closely  with  the  nature  of  those  tasks  to  promote  transfer 
of  skills  from  training  systems  to  the  theaters  of  operation.  Ultimately,  this  will 
mean  moving  from  desktop  training  environments  to  more  interactive  and 
physical  environments.  Research  will  be  needed  to  examine  a  learning 
progression  from  the  desktop  to  the  wild,  a  concept  where  Soldiers  can  receive 
training  anywhere.  We  examine  4  modes  of  adaptive  training  environments  to 
support  this  concept. 

Variable  task  dynamics  refer  to  the  physical  modes  of  interaction  of  the  learner 
during  the  training  experience.  This  ranges  from  static  (seated  position  for 
desktop  training),  to  limited  dynamic  (standing  position  limited  range  of  mobility 
in  instrumented  areas),  to  enhanced  dynamic  (standing,  kneeling,  and  prone 
positions  with  expanded  mobility  in  instrumented  areas),  to  “in-the-wild”  (any 
position  with  unlimited  mobility  where  sensors  and  communications  move  with 
the  Soldier). 

Variable  task  definition  refers  to  how  well  the  domains  are  understood  in  tenns  of 
standards  and  measures-of-perfonnance.  Well-defined  domains  (e.g., 
mathematics)  typically  have  1  correct  path  to  a  successful  outcome  and  a  set  of 
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specific  standards  for  measuring  success.  Ill-defined  domains  may  have  multiple 
paths  to  successful  outcomes,  and  they  tend  to  have  vague  standards  and  less 
defined  measures  of  success.  Ill-defined  domains  may  also  have  unexpected  and 
inconsistent  confounds  that  could  cause  learning  to  be  perceived  when  there  really 
is  a  mediating  underlying  factor.  Analyzing  these  can  provide  greater  knowledge 
to  answering  the  “why”  behind  performance  and  learning  outcomes. 

Finally,  task  complexity  refers  to  the  range  of  difficulty  in  understanding  and 
perfonning  the  task.  Task  complexity  can  range  from  simple  procedural  tasks  to 
more  complex  multidimensional  tasks. 

Next,  we  examine  modes  of  dynamic  interaction.  Limited  dynamic  environments 
support  hybrid  (i.e.,  cognitive,  affective,  psychomotor)  tasks  where  a  larger 
degree  of  interaction  with  the  environment  and  other  learners  is  critical  to 
learning,  retention,  and  transfer  to  the  operational  environment.  Decision-making 
and  problem-solving  tasks  may  be  taught  easily  in  a  limited  dynamic  mode  along 
with  tasks  requiring  physical  orientation  (e.g.,  land  navigation). 

Enhanced  dynamic  enviromnents  support  tasks  where  freedom  of  movement  and  a 
high  degree  of  interaction  with  other  learners  are  critical  to  learning,  retention,  and 
transfer  to  the  operational  enviromnent.  Building  clearing  and  other  team-based 
tasks  may  be  taught  easily  in  an  enhanced  dynamic  mode. 

“In-the-wild”  mode  is  transferring  tutoring  to  the  operational  environments  and 
could  also  be  called  embedded  training  for  Soldiers.  In-the-wild  mode  is  critical 
to  support  tasks  where  a  very  high  degree  freedom  of  movement  and  a  high 
degree  of  interaction  with  other  learners  are  critical  to  learning,  retention,  and 
transfer  to  the  operational  environment.  It  is  anticipated  that  psychomotor  and 
social  tasks  may  be  best  taught  in-the-wild  or  an  environment  more  closely 
resembling  the  operational  environment. 

7.  Interdependencies  with  Other  Adaptive  Training  Research 
Vectors 


This  section  examines  interdependencies  between  domain  modeling  and  the  other 
5  adaptive  training  research  vectors  (Fig.  4).  This  discussion  forms  the  basis  for 
the  sequencing  of  research  and  ultimately  bringing  adaptive  training  capabilities 
into  a  state  of  practice. 
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Adaptive  Training  Systems 


Fig.  4  Adaptive  training  research  vectors 

Accurate  methods  to  classify  individual  and  team  learner  states  are  a  necessary 
precursor  to  selecting  optimal  instructional  strategies  as  noted  in  the  learning 
effect  models  for  individual  learners  (Fig.  2)  and  teams  of  learners  (Fig.  3).  In 
turn,  instructional  strategies  along  with  instructional  context  are  necessary 
precursors  to  selecting  optimal  instructional  tactics  and  ultimately  significant 
effect  on  desired  outcomes:  learning,  perfonnance,  retention,  and  transfer. 

7.1  Learner  Modeling  and  Domain  Modeling 

Adaptive  training  systems  are  learner-centric  systems.  Independent  of  the  domain 
under  training,  accurate  modeling  of  the  learner  is  critical  to  driving  instructional 
decisions  in  adaptive  training  systems.  However,  collection  and  maintenance  of 
this  data  may  be  costly;  therefore,  it  is  necessary  to  select  measures  and  states  that 
significantly  impact  our  desired  outcomes:  learning,  perfonnance,  retention,  and 
transfer.  Research  is  needed  to  determine  what  this  dataset  will  look  like. 
Candidates  abound  in  the  literature,  but  in  general  these  include  transient 
data/states,  cumulative  states  (building  over  time),  and  enduring  data/states 
(Paneva  2006).  It  should  be  noted  that  learning  falls  into  both  transient  and 
cumulative  states.  It  is  necessary  to  understand  progress  toward  domain 
competency  in  addition  to  measures  of  near-term  perfonnance.  It  is  important  to 
understand  how  domain-specific  learning  (skills)  decay  over  time. 

Transient  measures  of  importance  include  individual  behavioral  and  physiological 
data,  and  cognitive,  affective,  physical,  and  social  states  to  represent  learning. 
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Cumulative  measures  include  achievements  (e.g.,  certifications,  training, 
education,  and  experiences),  affiliations,  work  history,  and  domain  competency. 
More  enduring  infonnation  about  the  learner  might  include:  gender,  culture,  first 
language,  physical  constraints  (e.g.,  colorblind/deaf),  values,  personality 
attributes,  or  other  trait-based  information  (Sottilare  and  Brawner  2014).  All  of 
these  measures/states  are  potential  drivers  for  adaptive  training  decisions. 

7.2  Automated  Instruction  and  Domain  Modeling 

In  GIFT,  instructional  management  takes  place  in  2  modules/processes  within  the 
learning-effect  model.  One  process  is  instructional  strategy  selection  within  the 
pedagogical  module.  The  second  is  within  the  domain  module  where  specific 
tactics  or  actions  are  selected  based  on  the  strategy  selection  and  instructional 
context.  Standard  representation  of  tactics  and  other  domain  knowledge 
components  are  needed  to  support  interoperability,  modularity,  and  reuse  of 
instructional  elements  to  reduce  the  cost  of  authoring  and  maintaining  adaptive 
training  systems. 

7.3  Authoring  Tools  and  Methods 

Authoring  tools  and  methods  are  needed  to  search,  retrieve,  curate,  and  apply 
domain  knowledge  to  adaptive  training  systems.  Automation  of  these  processes  is 
needed  to  reduce  the  overhead  of  gathering  and  organizing  domain  knowledge 
and  authoring  adaptive  training  systems.  Two  developing  automation  capabilities 
include  tools  to  rapidly  develop  expert  models  through  data  mining  of  text-based 
material  (e.g.,  field  manuals),  and  automated  generation  of  child  scenarios  from 
parent  scenarios.  Development  of  new  user  interfaces  for  authors  is  needed  to 
make  authoring  of  domain  knowledge  easier  for  authors  who  are  domain  experts, 
but  lack  computer  science  and  instructional  system  design  skills.  Finally,  adaptive 
training  systems  require  capabilities  to  interpret  user-generated  content  (e.g., 
social  media)  to  support  domain-specific  changes  (e.g.,  improved  accuracy  tactics 
selection  algorithms,  alternate  domain  content)  to  ensure  the  learner  population  is 
engaged  in  the  training  process. 

7.4  Evaluation 

Research  is  needed  to  evaluate  and  optimize  the  effect  of  tactics  selected  within 
the  GIFT  domain  module  upon  desired  outcomes:  learning,  perfonnance, 
retention,  and  transfer.  To  understand  the  impact  of  domain-specific  instructional 
decisions,  research  is  needed  to  provide  stealth  assessment  of  learners  (Shute  et  al. 
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2013).  Long-term  effects  (deep  learning,  retention,  and  transfer)  are  also 
important  in  evaluating  the  impact  of  adaptive  training  methods. 

7.5  Architecture 

GIFT,  as  the  architecture  to  facilitate  adaptive  training,  will  need  to  support  the 
acquisition  and  interpretation  of  data  required  by  the  domain  module  to  support 
optimal  instructional  tactics  selection  per  the  learning  effect  models 
(Figs.  3  and  4). 

8.  Conclusions 


This  report  outlines  the  ARL’s  plans  for  conducting  research  in  adaptive  training 
and  education  to  support  the  US  Army  Learning  Model.  Specifically,  this  report 
relates  to  domain  modeling  and  the  answer  to  the  question:  What  adaptive  training 
methods  provide  the  best  value  (in  terms  of  effectiveness  and  affordability)  for  the 
comprehensive  modeling  of  US  Army  Training  and  Education  Domains? 

This  report  outlined  goals  to: 

•  model  and  understand  the  efforts  required  to  author  domains  of  varying 
complexity,  definition,  and  physical  dynamics; 

•  support  authoring  of  ill-defined  and  well-defined  task  domains; 

•  support  authoring  of  militarily  relevant  training  and  education  domains 
across  the  spectrum  of  cognitive,  affective,  psychomotor,  and  social  tasks; 

•  match  the  nature  of  military  tasks  to  training  environments  and  optimize 
transfer  to  operational  environments; 

•  accurately  assess  learning  and  domain  task  performance  in  real-time; 

•  promote  optimal  learning,  performance,  retention  and  transfer  (on-the-job 
performance)  across  domains; 

•  support  individual  and  team  training  (e.g.,  small  unit  and  collective 
training)  and  education  (e.g.,  collaborative  learning  and  problem-solving) 
experiences. 
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